Using Audio Based Disambiguation for Improving Handwritten Mathematical Content Recognition in Classroom Videos

We consider the problem of recognizing handwritten mathematical content in classroom videos that capture the content written on the whiteboard and the content spoken by the instructor. While the problem of recognizing handwritten textual content from videos has been studied before, recognition of handwritten mathematical content and the use of audio content from classroom videos to assist in recognition, however, presents us with a new set of challenges. In this paper, we outline such challenges, and present the description of an end to end system that makes use of both video and audio based recognition components to improve the accuracy of handwritten mathematical content recognition. We have implemented the system using an open source implementation of a text recognizer and a commercially available phonetic word spotter. Preliminary results reported in this paper demonstrate the viability of our approach.